Skip to content

feat: packagist worker#4357

Merged
epipav merged 64 commits into
mainfrom
feat/packagist-worker
Jul 22, 2026
Merged

feat: packagist worker#4357
epipav merged 64 commits into
mainfrom
feat/packagist-worker

Conversation

@epipav

@epipav epipav commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

epipav added 13 commits July 16, 2026 13:52
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Copilot AI review requested due to automatic review settings July 16, 2026 17:26
@cursor

cursor Bot commented Jul 16, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Large new ingestion surface (~500K packages, external crawl) and significant growth in versions/package_dependencies; operational risk is mainly registry politeness, long drains, and correct watermark/audit behavior rather than auth or payments.

Overview
Adds a new packagist-worker Temporal service that ingests PHP/Composer packages from packagist.org (no deps.dev/BigQuery universe), documented in ADR-0009.

Data & workflows: Migration creates packagist_package_state with per-lane watermarks and run results. Four workflows run on dedicated crons (minutes off :00): weekly seed from list.json (then chain-starts metadata drain), metadata (dynamic stats + p2 manifests → versions, direct/dev dependency edges with constraints only, repos, critical-only maintainers), monthly downloads-30d for all packages, and daily downloads for the critical slice.

Implementation: New packagist/ module (fetch with polite UA/mailto, p2 minifier expansion, DAL reconciliation for package_dependencies), broad unit tests, Docker/scripts wiring, and CROWD_PACKAGES_PACKAGIST_* env vars. Security-contacts registry fetcher key changes from composer to packagist. canonicalizeRepoUrl gains GitLab subgroup and legacy deep-link handling for Packagist-declared repos.

Reviewed by Cursor Bugbot for commit 282c403. Bugbot is set up for automated code reviews on this repo. Configure here.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@epipav epipav changed the title Feat/packagist worker feat: packagist worker Jul 16, 2026
Comment thread services/apps/packages_worker/src/packagist/listPackages.ts Fixed
Comment thread services/apps/packages_worker/src/packagist/workflows.ts Outdated
@epipav
epipav requested a review from joanagmaia July 16, 2026 17:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a Packagist worker that discovers Composer packages and ingests metadata, dependencies, maintainers, repositories, and download statistics.

Changes:

  • Adds Packagist Temporal workflows, schedules, persistence, and deployment wiring.
  • Adds DAL queries and ingestion-state migration.
  • Adds documentation and unit tests.

Reviewed changes

Copilot reviewed 41 out of 43 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
services/libs/data-access-layer/src/packages/versions.ts Adds Packagist version upserts.
services/libs/data-access-layer/src/packages/packagistPackageState.ts Adds ingestion watermark queries.
services/libs/data-access-layer/src/packages/packages.ts Adds Packagist package DAL operations.
services/libs/data-access-layer/src/packages/index.ts Exports new DAL modules.
services/libs/data-access-layer/src/packages/dependencies.ts Adds dependency-edge upserts.
services/apps/packages_worker/vitest.config.ts Isolates Packagist tests.
services/apps/packages_worker/src/workflows/index.ts Exports Packagist workflows.
services/apps/packages_worker/src/security-contacts/extractors/registry/index.ts Registers Packagist contact extraction.
services/apps/packages_worker/src/scripts/triggerPackagistSeed.ts Adds manual workflow trigger.
services/apps/packages_worker/src/packagist/workflows.ts Defines ingestion workflows.
services/apps/packages_worker/src/packagist/upsertPackageInfo.ts Persists package information.
services/apps/packages_worker/src/packagist/upsertMetadata.ts Persists versions and dependencies.
services/apps/packages_worker/src/packagist/types.ts Defines Packagist types.
services/apps/packages_worker/src/packagist/schedule.ts Registers Temporal schedules.
services/apps/packages_worker/src/packagist/retryPolicy.ts Defines retry limit.
services/apps/packages_worker/src/packagist/README.md Documents worker behavior.
services/apps/packages_worker/src/packagist/normalize.ts Normalizes registry metadata.
services/apps/packages_worker/src/packagist/listPackages.ts Fetches and parses package catalog.
services/apps/packages_worker/src/packagist/fetchPackage.ts Fetches package endpoints.
services/apps/packages_worker/src/packagist/expandMetadata.ts Expands minified metadata.
services/apps/packages_worker/src/packagist/downloads.ts Persists 30-day downloads.
services/apps/packages_worker/src/packagist/activities.ts Implements Temporal activities.
services/apps/packages_worker/src/packagist/__tests__/wiring.test.ts Tests worker wiring.
services/apps/packages_worker/src/packagist/__tests__/persistMetadata.test.ts Tests metadata persistence.
services/apps/packages_worker/src/packagist/__tests__/normalize.test.ts Tests normalization.
services/apps/packages_worker/src/packagist/__tests__/listPackages.test.ts Tests catalog handling.
services/apps/packages_worker/src/packagist/__tests__/ingest.test.ts Tests ingestion behavior.
services/apps/packages_worker/src/packagist/__tests__/fetchPackage.test.ts Tests HTTP handling.
services/apps/packages_worker/src/packagist/__tests__/expandMetadata.test.ts Tests metadata expansion.
services/apps/packages_worker/src/packagist/__tests__/dueSelection.test.ts Tests due-selection queries.
services/apps/packages_worker/src/packagist/__tests__/downloads.test.ts Tests download windows.
services/apps/packages_worker/src/bin/packagist-worker.ts Adds worker entrypoint.
services/apps/packages_worker/src/activities.ts Exports Packagist activities.
services/apps/packages_worker/package.json Adds worker scripts.
scripts/services/packagist-worker.yaml Adds local service configuration.
scripts/builders/packages.env Adds worker build target.
docs/adr/README.md Indexes Packagist ADR.
docs/adr/0006-packagist-worker-design-decisions.md Records design decisions.
docs/adr/0005-pypi-downloads-bigquery-merge-scoping.md Expands decider name.
backend/src/osspckgs/migrations/V1783382400__packagist_worker.sql Creates ingestion-state table.
backend/.env.dist.local Adds local crawler contact setting.
backend/.env.dist.composed Adds composed crawler contact setting.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread services/apps/packages_worker/src/packagist/normalize.ts Outdated
Comment thread services/apps/packages_worker/src/packagist/upsertPackageInfo.ts Outdated
Comment thread services/apps/packages_worker/src/packagist/activities.ts Outdated
Comment thread services/apps/packages_worker/src/packagist/downloads.ts Outdated
Comment thread services/apps/packages_worker/src/packagist/activities.ts Outdated
Comment thread docs/adr/0006-packagist-worker-design-decisions.md Outdated
Comment thread docs/adr/0006-packagist-worker-design-decisions.md Outdated
Comment thread docs/adr/0006-packagist-worker-design-decisions.md Outdated
Comment thread services/apps/packages_worker/src/packagist/upsertMetadata.ts Outdated
Comment thread services/apps/packages_worker/src/packagist/downloads.ts Outdated
Copilot AI review requested due to automatic review settings July 17, 2026 09:00
Comment thread backend/src/api/public/v1/packages/getPackage.ts
Comment thread backend/src/database/repositories/segmentRepository.ts

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

epipav added 4 commits July 17, 2026 11:10
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Copilot AI review requested due to automatic review settings July 18, 2026 22:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

docs/adr/0009-packagist-worker-design-decisions.md:33

  • This new ADR does not follow the mandatory repository ADR structure. .claude/rules/adr-format.md:18-28 requires ## Decision, ## Alternatives Considered, and ## Consequences in that order (with Positive/Negative/Risks subsections), but this document uses a living ## Decisions layout and omits the alternatives section. Please restructure it to the enforced template or split the consolidated decisions into compliant ADRs.
## Decisions

Comment thread services/apps/packages_worker/src/packagist/activities.ts
Comment thread services/apps/packages_worker/src/packagist/upsertPackageInfo.ts Outdated
Copilot AI review requested due to automatic review settings July 18, 2026 22:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

docs/adr/0009-packagist-worker-design-decisions.md:33

  • This new ADR does not follow .claude/rules/adr-format.md: the mandatory heading is ## Decision (singular), followed by ## Alternatives Considered and ## Consequences with ### Positive, ### Negative, and ### Risks. Those required sections are absent or use a different structure, so please reformat the ADR before merging.
## Decisions

Comment thread services/apps/packages_worker/src/packagist/fetchPackage.ts Outdated
epipav added 3 commits July 19, 2026 01:05
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Copilot AI review requested due to automatic review settings July 19, 2026 10:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

services/apps/packages_worker/src/packagist/activities.ts:225

  • A successful response without a numeric downloads.monthly currently becomes a no-op and still advances downloads_30d_last_run_at. Because this lane has no backfill and the month is never revisited, a partial/malformed response permanently loses that month's observation. Treat the missing count as an error (so Temporal retries and eventually records an error) and only mark success after a numeric value is persisted.
  await persistPackagist30dWindow(qx, purl, info.value.package.downloads?.monthly ?? null, runDate)
  await markPackagist30dProcessed(qx, purl, { status: 'success', attempts: info.attempts })

services/libs/data-access-layer/src/packages/versions.ts:165

  • This cleanup can change versions.is_latest, but its row count is discarded, so changedFields may omit the only mutation (for example, when the incoming latest was already true while another stale row was also true). persistPackagistMetadata then writes no audit token for the cleanup. Capture the affected-row count and add versions.is_latest when any stale flag was cleared.
    docs/adr/0009-packagist-worker-design-decisions.md:33
  • This ADR does not follow the repository's mandatory ADR structure. .claude/rules/adr-format.md:18-28 requires ## Decision, ## Alternatives Considered, and ## Consequences with ### Positive, ### Negative, and ### Risks; this file uses ## Decisions, has no alternatives section, and embeds differently named consequence blocks per decision. Restructure it to the required template or update the governing rule before accepting the ADR.
## Decisions

services/apps/packages_worker/src/packagist/activities.ts:270

  • When downloads.daily is absent or non-numeric, this records the run as successful even though no row was inserted. The next schedule uses a new run date, so the missing day is never retried or backfilled. Treat the missing count as an error and only advance the daily watermark after a numeric count has been persisted.
  await markPackagistDailyProcessed(qx, candidate.purl, {
    status: 'success',
    attempts: info.attempts,
  })

Copilot AI review requested due to automatic review settings July 22, 2026 10:27
@epipav epipav self-assigned this Jul 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

docs/adr/0009-packagist-worker-design-decisions.md:33

  • This ADR does not follow the repository's mandatory ADR structure: .claude/rules/adr-format.md:18-28 requires ## Decision, ## Alternatives Considered, and ## Consequences with ### Positive, ### Negative, and ### Risks. The current consolidated ## Decisions layout omits those required sections, so it must be restructured or the rule must explicitly define an exemption for living ADRs.
## Decisions

Comment thread services/libs/data-access-layer/src/packages/versions.ts Outdated
epipav added 2 commits July 22, 2026 12:42
Signed-off-by: anilb <epipav@gmail.com>
Copilot AI review requested due to automatic review settings July 22, 2026 10:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

services/apps/packages_worker/src/packagist/activities.ts:339

  • This makes the nominal weekly refresh run approximately every two weeks. If run 1 starts at t0, its rows are stamped after t0; run 2 starts at t0 + 7d, computes dueCutoff = t0, and the strict metadata_last_run_at < dueCutoff predicate skips every row written during run 1. Please either select all rows last processed before the current run cutoff for the seed-triggered weekly sweep, or schedule the sweep more frequently and retain the seven-day rolling threshold.
  const dueCutoff = new Date(
    new Date(cutoff).getTime() - metadataRefreshDays() * 24 * 60 * 60 * 1000,
  ).toISOString()

docs/adr/0009-packagist-worker-design-decisions.md:33

  • This new ADR does not follow the repository's mandatory ADR structure. .claude/rules/adr-format.md:18-28 requires ## Decision, ## Alternatives Considered, and ## Consequences with ### Positive, ### Negative, and ### Risks; this document uses ## Decisions, has no alternatives section, and nests consequence labels differently. Please restructure it or update the ADR policy separately before adding this accepted ADR.
## Decisions

Comment thread services/apps/packages_worker/src/packagist/activities.ts
Copilot AI review requested due to automatic review settings July 22, 2026 10:58
@epipav
epipav merged commit 1987d18 into main Jul 22, 2026
14 checks passed
@epipav
epipav deleted the feat/packagist-worker branch July 22, 2026 11:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (3)

services/apps/packages_worker/src/packagist/activities.ts:339

  • With the default 7-day window and weekly seed schedule, this threshold skips every package processed after the previous run's start: its metadata_last_run_at = NOW() is later than this cutoff - 7 days. Early packages therefore refresh nearly every 14 days, not weekly. Record the fixed workflow cutoff as the watermark (or otherwise compare run anchors) so one run's completion timestamps are eligible on the next weekly run.
  const dueCutoff = new Date(
    new Date(cutoff).getTime() - metadataRefreshDays() * 24 * 60 * 60 * 1000,
  ).toISOString()

services/apps/packages_worker/src/packagist/activities.ts:138

  • A terminal NOT_FOUND only advances the state watermark, leaving a deleted Packagist package permanently status = 'active'. Because seeding is insert-only, nothing else will mark that row unpublished, and it remains in future crawls and downstream active-package data. Handle persistent NOT_FOUND by updating the package status while retaining the row for history.
    await markPackagistMetadataScanned(

docs/adr/0009-packagist-worker-design-decisions.md:33

  • This new ADR does not follow the repository's mandatory ADR structure: .claude/rules/adr-format.md:18-28 requires ## Decision, ## Alternatives Considered, and ## Consequences with ### Positive, ### Negative, and ### Risks. The plural ## Decisions plus inline consequence labels does not satisfy those headings; please restructure the document or update the governing rule first.
## Decisions


// Match: alpha|a|beta|b|rc followed by optional digits
// Stable: plain number, -patch\d*, -p\d*
if (suffix.match(/^(alpha|a|beta|b|rc)\d*$/)) {
})

const suffixMatch =
dashIdx === -1 ? null : v.slice(dashIdx + 1).match(/^(alpha|a|beta|b|rc)(\d*)/i)

let versionIds: Array<{ number: string; id: string }> = []
if (versionRows.length > 0) {
const versionResult = await upsertPackagistVersions(t, agg.id, versionRows, latestVersion)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants